Aircraft guidance systems are intended to display tactical and navigational information from a synthetic exterior view overlaid on an image reproduction device in the aircraft cockpit. Conventionally, perspective displays on cockpit display units have been confined to a limited segment of the airspace in the direction of the flight. This narrow angular range in front of the aircraft is not adequate for the tasks of the aircraft pilot, however, who must monitor all of the airspace. Until now only an area of observation disposed in a relatively narrow image angle in front of the aircraft has been displayed to the pilot on the image reproduction device in the cockpit.
This problem involving depicting an area of observation having a large image angle on an image reproduction device disposed in a cockpit relates not only to aircraft but also to other types of vehicles, for instance land vehicles or watercraft or even stationary devices.
U.S. Patent Application Publication 2009/0309811 A1 discloses an image display system in which an image to be displayed is compressed in the peripheral regions of the image before being displayed on a user's head-mounted display apparatus so that these peripheral regions that are in the user's peripheral vision are more compressed than the central image display area. This image that is compressed on the periphery is then re-expanded for depiction on the image reproduction device so that the image actually displayed is reproduced undistorted over the entire image angle, but has significantly lower resolution on the periphery. Accordingly, the number of pixels required for the image display is to be reduced so that the resolution of the displayed image is adapted to the user's vision. Thus, high image resolution is created in the central image area, while the image resolution in the peripheral visual range is low. The user thus obtains a high resolution image in the region of his central view, while the resolution of the image displayed for him decreases towards the peripheral regions of the image. It is also mentioned, however, that the re-expansion step may be omitted.
U.S. Pat. No. 5,495,576 A discloses generating a virtual image from a plurality of sensor data (e.g., radar, camera) that are detected simultaneously from all sides of a three-dimensional object, the virtual image being displayed to a user on a display apparatus.
U.S. Patent Application Publication No. 2007/0097206 A1 discloses a panoramic camera system that comprises a plurality of camera units attached to a vehicle, each of which is oriented in a different viewing direction around the vehicle. A panoramic image is obtained from image data recorded by these cameras and is reproduced on a display apparatus, enabling a 360° view around the vehicle.
Exemplary embodiments of the present invention are directed to an apparatus for rendering surroundings by means of which it is possible for an operator to see an area of observation at a wide image angle on an image reproduction device without distortions in the displayed image that could impair the operator's ability to apprehend the situation. Exemplary embodiments of the present invention also provide a vehicle having such an apparatus for rendering surroundings and to provide a method for depicting a panoramic image.
To this end the apparatus for rendering surroundings is provided with at least one image source, an image signal processing device, and at least one image reproduction device, the input image signals supplied by the image source being fed to the image signal processing device, the image signal processing device processing the input image signals fed to it to create output image signals for a panoramic image, the output image signals from the image signal processing device being forwarded to the image reproduction device for displaying the panoramic image, and the image signal processing device processing the input image signals such that at least a first region of the panoramic image is reproduced undistorted or compressed in a first compression and at least a second region of the panoramic image is reproduced compressed in a second compression. The image signal processing device is embodied such that the portion of the regions reproduced undistorted or compressed is variably selectable on the entire panoramic image, such that the compression of the regions reproduced compressed can be selected as desired, and such that these changes in the regions reproduced compressed may occur during the operation of the apparatus for rendering surroundings.
The image signal processing device in the inventive apparatus for rendering surroundings permits the image information produced by an image source or by a plurality of image sources, which information is fed to the image signal processing device as input image signals, to be processed such that the horizontal and/or vertical compression of the image angle that is covered by the image source or image sources at an image angle that corresponds to the view angle of the operator onto the image reproduction device is not compressed with the same compression over the entire surface of the panoramic image displayed on the image reproduction device, but rather is reproduced with different horizontal and/or vertical compressions.
One advantageous special case occurs when compression occurs only in one part of the panoramic image, while another part of the panoramic image, which as a rule corresponds to the main viewing direction of the operator, in which the operator must be able to see and recognize objects accurately, is reproduced in an undistorted manner.
In this manner it is possible for the operator, for instance the vehicle driver or the aircraft pilot, to have the region of his field of vision that is required for steering the vehicle or aircraft displayed correctly for him by the image reproduction device such that on the other hand, however, he can see peripheral to and where necessary above and below this main view direction on the image reproduction device a region that is compressed, that is, distorted, but in which the operator can perceive things that are outside of his natural field of vision.
While in the prior art the undistorted reproduction is always limited to the central view angle of the operator and the compressed image portions are provided in the peripheral region, in the inventive apparatus for rendering surroundings the portion of the second region reproduced compressed is variable and moreover the compression of the second region may be selected as desired. These changes in the second region reproduced compressed may be made as desired during the operation of the device for rendering surroundings.
This variability, which is not known from the prior art, makes it possible either for the operator himself to select the portion of the compressed region and the compression or, for example, when there is a threat coming from a specific direction (for instance, a missile that is approaching an aircraft and that is detected by a sensor on the aircraft), for the threat region to be automatically reproduced undistorted on the image reproduction device, even if this threat region is in a second region of the panoramic image that was previously reproduced compressed.
The image signal processing device is preferably embodied such that at least one additional region of the panoramic image may be reproduced with another compression that is different from the first and second compressions. This embodiment makes it possible for the panoramic image to be compressed by region as desired with different compressions. For instance, regions of the panoramic image that are less important for a current operating situation may be reproduced highly compressed and regions that are less important for the current operating situation may be reproduced less highly compressed or even undistorted. These regions may be selected either manually by the operator or automatically by a corresponding navigation or vehicle guidance support device. The selection of the regions to be compressed and the selection of the compression to be assigned to each respective region occurs during the running time so that in each operating situation it is possible to adjust an individual depiction of the panoramic image manually or automatically.
In another advantageous embodiment, the sight angle corresponding to a region reproduced undistorted or compressed may be selected as desired in a horizontal plane between 0° and 360° and also the sight angle corresponding to a region reproduced undistorted or compressed may be selected as desired in a vertical plane between the nadir sight angle of −90° and the zenith sight angle of +90°. In this manner any desired selection of the sight angular range to be compressed and to be displayed on the panoramic image screen may be made from the entire visual range surrounding the operator or the specific sensor position.
Preferably a plurality of image sources is provided, the viewing directions of the image sources being oriented in different directions about an operator, preferably proceeding from the operator observing the surroundings. In this manner it is possible for the apparatus for rendering surroundings to cover a very wide observation angle.
In one preferred embodiment, at least one of the image sources has a data processing device that is programmed to transmit to the image signal processing device image data of a stored digital terrain model as image signals. This feeding of image data of a stored digital terrain model as a realistic image of a virtual view in a specific viewing direction can occur by means of a computer without additional image source hardware. The use of data from a stored digital terrain model as image signals has a number of advantages over obtaining image signals by means of sensors in real time. The data stored in a digital terrain model can be significantly more precise because they may be obtained for instance with measuring methods that are more complex than is possible in real time using onboard means. Moreover, the use of image data from a stored digital terrain model does not depend on the weather. Also, the data from the terrain model are not subjected to any shadowing, as may be the case when obtaining data in real time by means of sensors, for instance due to terrain structures in the area to be covered by an image sensor. While sensors for obtaining image data in real time are subject to a range limitation, there are no range limitations when using image data from a stored digital terrain module if the user is moving in the stored terrain model. Another important advantage of the use of image data from a stored digital terrain model compared to obtaining image data by means of active sensors, that is, sensors that radiate a signal, is the fact that access to the image data from the stored digital terrain model is “silent,” which is important especially during military operations to prevent the user from being detected by a signal being radiated by an active sensor.
At least one of the image sources is preferably formed by at least one sensor that is configured to detect approaching objects and to transmit to the image signal processing device as image signals position data for this object, the position data resulting from the detection. With this variant, sensors that detect, for instance, approaching flying objects in non-visible light (radar or infrared sensors), may be integrated into a total system comprising a plurality of image sources. It is especially possible to obtain on the image reproduction device a reliable rendering of an object approaching the operator while attaining the aforesaid advantages of the digital terrain model when these sensor signals are overlaid on a digital terrain model.
In another preferred embodiment, at least one of the image sources is formed by an image acquisition device, for instance a video camera. This variant is especially suitable when the region located in the viewing direction of the observer, for instance a segment disposed in front of a vehicle, is to be integrated into the panoramic image as a live image for display on the image reproduction device.
The viewing directions for the image sources are preferably oriented such that the image sources include, at least in a first plane image, information for a field of vision with an image angle of greater than 120°, preferably greater than 180°, more preferably 360°. The horizontal expansion of the field of vision for both eyes of a person extends approximately 180°, perception of moving objects being reduced, however, at the outer periphery and specific object recognition no longer being possible at the peripheral area of the field of vision. The inventive apparatus for rendering surroundings clearly supplies more information particularly when the horizontal field of vision that is covered by the image sources or by the image source is between 120° and 180° since not only can the operator just perceive objects disposed in the peripheral regions of his field of vision due to their movement, but he can also recognize and possibly identify them.
It is particularly advantageous when the region of the panoramic image that corresponds to the viewing direction of the operator is reproduced undistorted. This makes it possible for the operator to always be able to see clearly and undistorted the region that is important to his need for information.
The image signal processing device preferably obtains information about the current viewing direction of the operator so that the region of the panoramic image that is reproduced undistorted corresponds to the current viewing direction of the operator, even if the operator moves and thus changes his viewing direction. Such automatic adjustment of the region reproduced undistorted to the current viewing direction of the operator makes it possible to allow the region of the panoramic image reproduced undistorted and displayed on the image reproduction device to shift when the operator changes his viewing direction. Accordingly, it is not necessary for the entire viewing angle that corresponds to the natural field of vision of the operator to be reproduced undistorted, but rather only a central, narrow viewing angle from the image reproduction device needs to be reproduced undistorted.
However, it is also advantageous when the operator can influence the image signal processing device such that it can select regions of the panoramic image displayed for him on the image reproduction device that are to be reproduced undistorted. In this manner it is possible for the operator to view undistorted an object discovered in the peripheral region of the panoramic image, which is reproduced distorted, by shifting the region of the panoramic image that is reproduced undistorted or by creating an additional region that is reproduced undistorted.
The apparatus for rendering surroundings is preferably part of a vehicle, especially an aircraft, and the image source is or the image sources are arranged on the vehicle and preferably provide a rendering that proceeds from the vehicle and that is at least semi-circular or circular. This embodiment of the inventive apparatus for rendering surroundings is especially advantageous for many areas of application in which what happens in the peripheral sections of a pilot's natural field of vision can be significant, with relatively low complexity, because it provides a noticeable gain in information.
The undistorted region of the panoramic image preferably reproduces an image sector located in the direction of travel. This makes it possible for the vehicle driver to also drive the vehicle exclusively using the panoramic image displayed on the image reproduction device, which represents a great advantage for instance for armored vehicles or even for remotely controlled vehicles for which the pilot and the image reproduction device are disposed outside of the vehicle.
In another preferred embodiment, the compression of the image reproduction increases from the region of the panoramic image that is reproduced undistorted to its sides. Because of this, the region of the panoramic image that is less relevant for the current main viewing direction takes up less space on the image reproduction device so that the remaining space can be better used for the regions of the panoramic image that are reproduced less distorted or undistorted.
One particularly advantageous use of the apparatus for rendering surroundings in accordance with the present invention is in a vehicle, especially in an aircraft, that is equipped with an inventive apparatus for rendering surroundings that may also have the aforesaid advantageous refinements.
The apparatus for rendering surroundings is used in a particularly advantageous manner when the vehicle is an unmanned vehicle, such as an unmanned aircraft. In this case the image reproduction device is provided in a remote control unit provided outside of the vehicle. The image signal processing device may be provided in the vehicle or in the remote control unit. The corresponding image signals are transmitted, preferably wirelessly, between the vehicle and the remote control unit.
The part of the object that relates to the method is attained using the method for depicting a panoramic image for an operator on an image reproduction device of an inventive apparatus for rendering surroundings. This method has the following steps:                Providing input image signals that are supplied by at least one image source;        Processing the input image signals to create output image signals for the panoramic image such that at least a first region of the panoramic image is undistorted or is compressed in a first compression and at least a second region of the panoramic image is compressed in a second compression.        Forwarding the output image signals to the image reproduction device, and,        Displaying the panoramic image on the image reproduction device, each compression of the regions reproduced compressed being individually variably selectable as desired.        
The processing of the input image signals in accordance with the inventive method attains output image signals produced from the input image signals obtained from the image source or sources, and these output image signals display on the image reproduction device a panoramic image that is reproduced with different distortion in different regions, it being possible to adjust the distortions, especially compressions, in one or two dimensions during the display, that is as desired during running time.
In one advantageous refinement of this method, an operator may select where an undistorted or distorted reproduction should occur on the panoramic image. This can occur, for instance, by the operator touching the region that is to be undistorted or distorted on the image reproduction device embodied as a touch-sensitive image screen or the operator can use operating means, known per se, on the image reproduction device to place a cursor on the desired location at which the undistorted or distorted image reproduction is to occur.
Alternatively or in addition thereto, it may also be provided that a region to be reproduced undistorted or less distorted on the panoramic image displayed on the image reproduction device is determined automatically based on an automatically detected object or event. Thus, for instance, when a sensor detects an object approaching from a specific direction, the region in which this object is located may be automatically reproduced undistorted or less distorted on the image reproduction device.
Preferred exemplary embodiments of the invention with additional details and further advantages are described and explained in detail in the following and reference the attached drawings.